Skip to content

docs: add -DCMAKE_BUILD_TYPE=Release for TurboQuant support#1

Merged
spiritbuun merged 1 commit into
spiritbuun:feature/turboquant-kv-cachefrom
chrisqianz:feature/turboquant-kv-cache
Mar 28, 2026
Merged

docs: add -DCMAKE_BUILD_TYPE=Release for TurboQuant support#1
spiritbuun merged 1 commit into
spiritbuun:feature/turboquant-kv-cachefrom
chrisqianz:feature/turboquant-kv-cache

Conversation

@chrisqianz

Copy link
Copy Markdown
Contributor

Overview

Additional information

Requirements

@spiritbuun
spiritbuun merged commit c70f52d into spiritbuun:feature/turboquant-kv-cache Mar 28, 2026
spiritbuun pushed a commit that referenced this pull request Apr 6, 2026
Codex post-commit review found:
1. TURBO_D was QK_TURBO3 (now 32) — broke turbo4 C array sizes
2. SET_ROWS kernel turbo3-specific but instantiated for turbo4
3. Tail block drop for non-128 head dims

Fixed #3 (TURBO_D). #1 and #2 don't affect turbo3+dk128 path.

Co-Authored-By: tturney@psyguard.ai
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
spiritbuun pushed a commit that referenced this pull request Jul 12, 2026
…n 2nd request

Closes #71. Reported by @TheLeung who also identified the root cause and
proposed this fix independently.

When --mmproj-gpu-swap is enabled, swap_mtp_to_mmproj_gpu() destroys ctx_dft
(the MTP draft context) to free VRAM for GPU-side image encoding. After the
image is processed, swap_mmproj_to_mtp() was supposed to recreate it, but
checked "if (ctx_dft)" — which is always nullptr since we just destroyed it in
the previous step. The entire recreation block was skipped.

After request #1 completes: ctx_dft is nullptr, all slots have slot.ctx_dft =
nullptr, and spec/spec_shared are null. On request #2, prompt cache save/load
attempts to access draft state through the null ctx_dft, causing either a
GGML_ASSERT(backend) crash or segfault during "updating prompt cache".

Fix: save whether MTP was active before destroying ctx_dft using
mtp_was_active_before_swap, then check that flag instead of the already-null
pointer in swap_mmproj_to_mtp(). This correctly recreates the MTP context,
speculative state, and slot bindings after every swap cycle.

The original implementation (commit 8e64d7a) always recreated MTP
unconditionally; a later refactor added the if (ctx_dft) guard to handle the
case where no MTP was configured, but forgot that ctx_dft is already destroyed
by the time swap_mmproj_to_mtp() runs.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants